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SENSOR AND IMAGING SYSTEM 
DESCRIPTION 

Related Applications 

This application claims the benefit of U.S. Provisional Patent Application 
Serial No. 60/229,613, filed August 31, 2000, and incorporated herein by reference. 
Technical Field 

The present invention relates to sensor and imaging systems, and more 
particularly to a system for providing and interpreting image data 
Background of the Invention 

Sensorandimaging systems aremcreasinglyindemandintoday'stechnology 
driven economy. These systems include a camera for viewing objects included within 
a field of view. The camera generates image data that is analyzed by a computer to 
determine what, if any, action should be taken in response to the object detected. 

Many recognition systems use two or more cameras for viewing objects 
included within one field of view. In addition to the costs associated with using several 
cameras, these systems require a specific mounting arrangement for each of the cameras. 
Such systems have a reduced level of reliability over single camera systems because both 
cameras are needed for proper operation. 

Single camera systems are typically mounted at a fixed location and lookfor 
objects that satisfy, or fail to satisfy, predetermined criteria For instance - systems that 
check for structural defects. These systems are incapable of making decisions that are 
20 not already specified. 

Accordingly, a need exists for a sensor and imaging system that, by using an 
image provided by a camera, can decide whether a condition has, or has not, been 
satisfied. 
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Summary of the Invention 

In an embodiment in accordance with the present invention, a system is 
provided having a camera, aprocessor, and a user interface. The camera transmits image 
data responsive to a scene within a field of view. In response to the image data, the 
5 processorindicates whether a condition has been satisfied. The user interface is operably 
connected to the processor and allows a user to select criteria for detection of objects, 
for indicating criteria selected, and for providing visual confirmation that an object has 
been detected. 

Inanomerembodiment,acontrolmterfaceisdsoprovidedforeffectmgother 
10 devices. Further, the system provides signals to influence other devices. 

In yet another embodiment, the system provides a signal to open a door upon 
a determination by the processor that a condition has been satisfied. The door is then 
open by a conventional electro mechanical door opener system having a drive motor 
operably connected to the door. 

Other features and advantages of the invention will be apparent from the 
following specification taken in conjunction with the following drawings. 
Brief Description of the Drawing s 

FIGURE 1 is a block diagram of an embodiment of a sensor and imaging 
system in accordance with the present invention; 
20 FIGURE 2 is a simplified schematic of another embodiment of a sensor and 

imaging system in accordance with the present invention and having a pair of camera 
assemblies connected to a pair of sensor/imaging circuits, respectively; 

FIGURE 3 is an elevation view of aclosed door having the camera assemblies 
of FIGURE 2 mounted in proximity thereto; 
25 FIGURE 4 is similar to FIGURE 3 except the door is open; 

FIGURE 5 is a perspective partial assembly view of one of the camera 
assemblies of FIGURE 2 for attachment to a header above the door of FIGURE 3; 

FIGURE 6 is a cross-sectional view of the camera assembly of FIGURE 5; 
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FIGURE 7 is a plan view of a heater assembly attached to the window of the 
camera assembly housing of FIGURE 5; 

FIGURE 8 is a cross-sectional view of the heater assembly, and window, 
taken along plane A-A of FIGURE 7; 
5 FIGURE 9 is a simplified schematic of the camera within the camera 

assembly of FIGURE 8 connected to one of the video processors of FIGURE 2, 

FIGURE 1 0 is a top, perspective view, of one side of the door of FIGURE 3 
and 4 wherein one of the camera assemblies of FIGURE 2 has a field of view that 
includes the area in firont of the door; 
10 FIGURE 1 1 is a graphical illustration of the pixel density of the field of view 

within FIGURE 10 as objects are placed further away from the camera assembly; 

FIGURE 12 provides a plurality of safety zone configurations that can, in an 
embodiment, be selected by a user using the membrane keypad of FIGURE 2; 

FIGURE 13 is a plan view of an embodiment of the membrane keypad 
15 depicted in FIGURE 2; 

FIGURE 14 is a plan view of a zone positioning method using the membrane 
keypad of FIGURE 13; 

FIGURE 1 5 is a simplified block diagram of one of the object sensor/imaging 
circuits of FIGURE 2 having a Field Programmable Gate Array (FPGA) and a Central 
20 Processing Unit (CPU); 

FIGURE 16 is a top-level simplified block diagram of a processing system 
in accordance with the present invention, having an image processing module within the 
FPGA of FIGURE 1 5, a feature processing module executed by the CPU of FIGURE 1 5, 
and a detection processing module executed by the CPU of FIGURE 15; 
25 FIGURE 17isasimplifiedblockdiagramofaninitializationsequenceforthe 

processing system of FIGURE 16, for initializing input parameters and calculating 
related, derived parameters, for initializing detection threshold tables, and for initializing 
detection and feature calculation zones; 
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FIGURE 18 is a simplified block diagram of the initialization sequence for 
the processing system of FIGURE 16, for initializing the FPGA image processing 
thresholds and video digitizer settings from user , input data via the FPGA serial 
input/output board, and for initializing the feature and detection processing systems; 

FIGURE 19 is a simplified block diagram of a video loop processing 
sequence for the processing system of FIGURE 1 6, utilizing a ping-pong buffer to point 
to, load and unpack reference images into user data; 

FIGURE 20 is a simplified block diagram of the video loop processing 
sequence for the processing system of FIGURE 16, utilizing user data in the CPU to 
generate features and detection decisions on a current image frame; 

FIGURE 21 is a simplified block diagram of the video loop processing 
sequencefortheprocessingsystemofFIGURE 16, having diagnostic messages outputted 
at the end of each frame processing and at any point in the video processing; 

FIGURE 22 is a simplified block diagram of a ping-pong system in 
1 5 accordance with the present invention; 

FIGURE 23 is a simplified block diagram of the initialization of the ping- 
pong system depicted in FIGURE 22; 

FIGURE 24 is a.simplified block diagram of the ping-pong process loop for 
the ping-pong system of FIGURE 22; 

FIGURES 25 is a simplified block diagram of an automatic contrast 
compensation initialization for the processing system of FIGURE 16, for changing the 
video gain in response to image characteristic criteria and time dynamic criteria; 

FIGURE 26 is a simplified block diagram of the automatic contrast 
compensation in a video loop for the processing system of FIGURE 16, for changing the 
video gain in response to image characteristic criteria and time dynamic criteria; 

FIGURE 27 is a simplified block diagram of the zone initialization sequence 
forthe system initialization of FIGURE 17, for buHding zones in real world coordinates, 
generating zone submasks and constructing zone masks; 
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FIGURE 28 is a simplified block diagram of the threshold tables initialization 
sequence for the system initialization of FIGURE n.havinganiiiitializecameraintrinsic 
parameters module, a resolution model module and an object model module; 

FIGURE 29 is a simplified block diagram of the image processing module for 
the processing system of FIGURE 16, having an edge detector; 

FIGURE 30 is a simplified block diagram of the image processing module for 
the processing system of FIGURE 16, having a motion detector that uses regions; 

FIGURE 3 1 is a simplified block diagram of the image processing module for 
the processing system of FIGURE 16, having region analysis for shadow and lightbeam 
processing; 

FIGURE 32 is a simplified block diagram of the image processing module for 
the processing system of FIGURE 16, having a motion detector that uses edges; 

FIGURE 33 is a simplified block diagram of the feature processing module 
for the processing system of FIGURE 16, for calculating presence, motion, frame and 
15 region features; 

FIGURE 34 is a simplified block diagram of the feature generation system of 
FIGURE 33, having a label module, a calculate global presence features module and a 
calculate zone presence features module; 

FIGURE 35 is a simplified block diagram of the calculate presence feature 
20 system of FIGURE 34, for calculating global presence features; 

FIGURE 36 is a simplified block diagram of the calculate presence feature 
system of FIGURE 34, for calculating zone presence features; 

FIGURE 37 is a simplified block diagram of the feature generation system of 
FIGURE 33, having a label module, a calculate global shadow and lightbeam features 
25 module, and a calculate shadow and lightbeam zone features module; 

FIGURE 38 is a simplified block diagram of the calculate shadow and 
lightbeam region features system of FIGURE 37, for calculating global shadow and 
lightbeam features; 
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FIGURE 39 is a simplified block diagram of the calculate shadow and 
lightbeam region features system of FIGURE 37, for calculating shadow and lightbeam 
zone features; 

FIGURE 40 is a simplified block diagram of a split histogram grey level 
analysis for the processing system of FIGURE 16; 

FIGURE 4 1 is a simplified block diagram of the feature generation system of 
FIGURE 33, for calculating frame features; 

FIGURE 42 is a simplified block diagram of the information processing 
module of FIGURE 16, for detecting the presence of an object from features generated; 

FIGURE 43 is a simplified block diagram of the information processing 
module of FIGURE 16, for evaluating and updating reference images; 

FIGURE 44 is a simplified block diagram of the information processing 
module of FIGURE 16, for changing threshold values relative to changing background 
values from the field of view; 

FIGURE 45 is a simplified block diagram of the information processing 
module of FIGURE 16, for determining the geometric association of edge and zone data 
in a detected object; 

FIGURE 46 is a continuation of FIGURE 45 for the simplified block diagram 
of the information processing module of FIGURE 16, and includes testing region edges 
20 to determine zone intersections in detected objects; 

FIGURE 47 is a continuation of FIGURE 46 for the simplified block diagram 
of the information processing module of FIGURE 16, and includes evaluating region 
scores of zones to determine zone intersections in detected objects; 

FIGURES 48 and 49 are tables for defining various image operations; 
FIGURE 50 is a representative example of a background or reference image; 
FIGURE 5 1 is similar to FIGURE 50 except it is a current image wherein an 
object has entered the field of view; 
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FIGURE 52 is the difference between subtracting the reference image in 
FIGURE 50 from the current image in FIGURE 51; 

FIGURE 53 is the difference between subtracting the current image in 
FIGURE 5 1 from the reference image in FIGURE 50; 

FIGURE 54 is a resulting image after thresholding and shape filtering the 
image in FIGURE 52; 

FIGURE 55 is a resulting image after thresholding and shape filtering the 
image in FIGURE 53; 

FIGURE 56 is a resulting image after completing a logical OR operation on 
the images in FIGURE 54 and FIGURE 55; and 

FIGURE 57 is similar to FIGURE 56 except that regions within the image of 
FIGURE 56 are labeled for classification. 
Detailed Description of the Preferred Embodiment 

While this invention is susceptible of embodiments in many different forms, 
there is shown in the drawings and will herein be described in detail preferred 
embodiments of the invention with the understanding that the present disclosure is to be 
considered as an exemplification of the principles of the invention and is not intended 
to limit the broad aspect of the invention to the embodiments illustrated. 

Turning to FIGURE 1, a block diagram of an embodiment of a system in 
accordance with the present invention is depicted. The system 10 includes aspects 
directed to image analysis 12, presence detection 14, sensor technology 16, electronics 
18, device control 20, camera pose 22, camera technology 24, image analysis 26, and 
market applications 28. 

Turning to FIGURE 2, a simplified schematic of another embodiment of a 
system in accordance with the present invention is depicted. The system 110 includes 
a pair of camera assemblies 112,113 operably connected, respectively, to a pair of 
sensor/imaging circuits 114,115. The sensor/imaging circuits 114,115 are operably 
connected to an I/O display board 1 1 6 that is connected to a keypad 118. In a further 
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embodiment, the system 1 10 can include an electro mechanical door opener system 1 17 
having a drive motor 1 19 operably connected to a door 121 (FIGURES 3 and 4). 

The cameras assemblies 112 andll3 can include charge coupled devices 
(CCD), or the like, having preferably a wide-angle lens, and capable of transmitting 
5 image data to the sensor/imaging circuits 1 14 and 1 15, respectively. The image data 
corresponds, respectively, to a scene within each camera's field of view. 

The sensor/imaging circuits 1 14,1 15 process the image data for determining 
whether a user selected condition has been satisfied. The user selected conditions are 
selected via a man-machine interface comprising the I/O display board 116 and the 

10 membrane keypad 118. In an embodiment, the man-machine interface is operably 
connected to the sensor/imaging circuits 1 14,1 15 and allow a user to select criteria for 
detection of objects, for indicating criteria selected, and for providing visual confirmation 
that an object has been detected. 

FIGURE 3 provides a cross sectional elevation view of a doorway 120 that 

15 is closed by a door 121. Mounted in proximity to the doorway 120 and on opposite sides 
thereof are the camera assemblies 112,113 of FIGURE 2. According, the camera 
assemblies 1 12 and 1 13 haveafield-of-view 122 and 123, respectively, on opposite sides 
of the doorway. Preferably, the field-of-view of at least one camera assembly includes 
the doorway 120 when the door 121 is open as shown in FIGURE 4. Further, the field- 

20 of-views 122 and 123 overlap about the doorway 120. 

Turning to FIGURE 5 a perspective partial assembly view is depicted of one 
of the camera assemblies 112,113 of FIGURE 2 for attachment to a header above the 
door 121 of FIGURE 3. The camera assembly includes a housing assembly 131 having 
a window 132. FIGURE 6 provides a cross-sectional view of the camera assembly of 

25 FIGURES. Mounted within the housing assembly 131 is a camera 134 having a field-of- 
view 136 and a heater assembly 140. In a further embodiment, a power supply (not 
shown) can be mounted within the heater assembly for supplying power to the camera 
134. 
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FIGURES 7 and . 8 provide a plan view and a cross-sectional view, 
respectively, of the heater assembly 140. In an embodiment, the heater assembly 140 
adjoins the window 132 of the camera assembly housing 131, preferably made of cast 
metal or the like, and includes a ceramic resistor 142 sandwiched between a pair of 
conductive terminals 144,145. The ceramic resistor 142 is generally ring shaped and 
constructed of a material that exhibits what is commonly referred to as a PTCR (positive- 
temperature coefficient or resistance) or PTC effect. Likewise, the conductive terminals 
144,145 are generally ring shaped and constructed of an electrically and thermally 
conductive material such as copper. 

In an embodiment, annular apertures 146 extend through the axis of the 
ceramic resistor 142 and the conductive terminals 144,145. The apertures 146 have 
substantially identical outer circumferences and are concentrically aligned with each 
other. 

Preferably, the outer perimeter 148 of conductive terminal 145 includes a 
plurality of ears 150 extending outwardly therefrom. Extending through each ear 150 is 
an aperture 1 52 for extending an attachment screw 154 (FIGURE 9) therethrough. 

TurningtoFIGURE9,theheaterassembly 140 is mounted within the housing 
131 ofthecameraassembly. As indicated previously, attachment screws 154 couple the 
conductive terminal 145 of the heater assembly 140 to a mounting surface, or 
alternatively, mounting posts 156 that inwardly extend into the housing 131 and are 
integrally attached thereto. 

The camera assembly housing 131 includes an aperture 158 that allows the 
camera's field-of-view 136 to extend outside of the housing. The window 132 is 
mounted over the aperture 158 to prevent contaminants such as dirt and moisture from 
25 entering the camera assembly. 

Preferably, the window 1 32 is sandwiched between the thermally conductive 
terminal ring 145 of the heater assembly 140 and an annular gasket 160 made of a 
resilient material and adjoining against the inner surface 162 of the camera assembly 
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housing 131 about aperture 158. In an embodiment, the window 132 is made of a 
visually transparent material such as borosilicate glass 

The camera 134 generates image data or electronic pixel data 218 
representative ofwhat is observed in the camera's field-of-view 136. In an embodiment, 

5 the image data 218 is analyzed by one of the video processors 1 14,1 15 (FIGURE 2) for 
determining what, if any, action should be taken in response to what is detected in the 
camera's field-of-view 136. 

Within the camera assembly housing 131, the terminals 144,145 ofthe heater 
assembly 140 are coupled to a voltage source 166 for maintaining a voltage potential 

10 across the ceramic resistor 142. The heat generated by the ceramic resistor 142ascurrent 
flows therethrough is dissipated through conductive terminal 145 and onto the window 
132. In an embodiment, enough heat is provided to maintain the window 132 at a 
temperature above the dew-point ofthe air outside ofthe camera assembly housing 13 1 . 
For instance, the heater can activate at about 87 °F and deactivate at about 106°F. 

15 As will be appreciated by those having skill in the art, the use of a ceramic 

resistor 142 eliminates the need for a mechanical thermostat or the like since the resistor 
material exhibits a positive-temperature coefficient. The elimination of a thermostat 
increases the reliability of the heater and reduces the amount of noise placed on voltage 
supplies as a result of switching. 

20 TurningbacktoFIGURE2,inanembodimentthecameraassemblies 112,113 

can include an illumination source (not shown), such as a lightbulb, light emitting diodes 
within the humanly visible or non-visible spectrum, or the like, to illuminate the field of 
view. The output of the camera assemblies 1 12,1 1 3 can be received by a removably 
coupled display (not shown) for allowing a user to view check that each camera 

25 assemblies' field of view is properly configured. 

Turning to FIGURE 10 a top perspective view is provide of one side of the 
door 121 ofFIGURE 3 and 4 wherein one ofthe camera assemblies (in particular camera 
assembly 112) ofFIGURE 2 has a field of view 122 that includes the area in front ofthe 
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door. The camera assembly 122 provides image data to the video processor 114 
(FIGURE 2) which, in an embodiment, superimposes a safety zone 168 and, if desired, 
one or more activation zones 170. Preferably, when an object is detected by either of the 
video processors 1 14,1 15 to be in a safety zone 168, the door 121 is opened and remains 

5 so until the object is outside of the safety zone 168. Further, when an object is detected 
by either of the video processor 1 1 4, 1 1 5 to enter an activation zone 1 68, the door 1 2 1 is 
opened are remains open for a period of time after the object stops moving in the 
activation zone or leaves the activation zone. 

Preferably, the safety zone 168 is maintained in an area immediately 

10 surrounding the door 121 to prevent the door from closing when a person or object is in 
the immediate vicinity of the door. Moreover, the activation zone(s) 168 open the door 
when a person or vehicle approaches the door 121. A failsafe system can also be 
provided to open the door 121 whenever there is a loss of illumination within the field 
of view, severe illumination changes, electronics failure, camera knocked ajar, or the 

1 5 camera lens is obscured. 

FIGURE 1 1 depicts the pixel density of the field of view 122 as objects are 
placed further away from the camera assembly 122. FIGURE 12 provides a plurality of 
safety zone configurations that can, in an embodiment, be selected by a user using the 
membrane keypad 118 (FIGURE 2). 

20 FIGURE 13 provides an plan view of an embodiment of the membrane 

keypad 118 depicted in FIGURE 2. The membrane keypad 118 and the I/O display board 
116 (FIGURE 2) provide a user interface or man-machine interface for a user to setup 
the system (FIGURE 2). In an embodiment, the user interface allows a user to define the 
door type, and dimensions, and establish the size, location, and type of detection zones 

25 the system 110 will use. Preferably, the user interfaces allows a user to enter data to 
define at least one control zone parameter (i.e., activation zone or safety zone) from the 
group consisting of area, location, shape, number of control zones, and control criteria. 
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As indicated previously, the system 110, via the user interface, has the 
capability to define at least a portion of an image received by the camera assemblies as 
a control zone. In an embodiment, the system 110 has the capability to choose 
coordinates from all pixel coordinated by direct access within the control zone. 
5 Alternatively, the system 110 has the capability to choose from multiple predefined 
zones. Moreover, the system 1 10 can have the capability to put real objects in the field 
of view so as to delineate boundary coordinates and the real objects become part of the 
image data. 

In an embodiment, the user interface has three modes of operation: parameter 

1 0 edit mode, run mode, and diagnostic mode. In parameter edit mode, a user can input or 
modify configuration parameters, using touch keypad buttons 610, 622, 624, and 626, 
such as the door model, English or metric units, camera heights and distance from the 
door. In the run mode, the system 1 10 is activated. As such, the system 110 processes 
images from the cameras 112,113 and outputs safety and activation zone indication 

1 5 signals through the I/O board 116, and displays status information on the display LEDs 
614, and 616. In the diagnostic mode, additional information regarding the status of the 
system 1 10 is made available via an I/O port (not shown). 

FIGURE 14 provides a diagram depicting movement of zones, using the 
membrane keypad of FIGURE 1 3 . In an embodiment, a user can move activation zones 

20 to various locations within the field-of-view of the camera assemblies 112,113. 

FIGURE 1 5 provides a simplified block diagram of one of the sensor/imaging 
circuits 114,1 15 ofFIGURE2. In an embodiment, the sensor/imagining circuits 114,115 
are substantially similar in physical design and include aField Programmable Gate Array 
(FPGA) 124, a Central Processing Unit (CPU) 125, and a video digitizer 126. 

25 In an embodiment, the video digitizer 126 receives an analog image signal 

from one of the cameras, digitizes the analog image signal, and transmits the digitized 
image signal to the field programmable gate array 124. 
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As explained in detail further herein, the field programmable gate array 124 
is programmed to perform one or more image processing operations in response to the 
digitized image signal received. In an embodiment, these operations include comparing 
predetermined traits of the digitized image signal with one or more previously received 
digitized image signals to provide composite image data. In response to the composite 
image data, the central processing unit 125 determines whether one or more conditions 
have been satisfied. 

Operably coupled to the video digitizer 126 is a temperature sensor 128 
having an output correlating to the temperature of the video digitizer. Upon an indication 
by the temperature sensor of a condition whereby the video digitizer 126 is not operating 
within a predetermined temperature range or limit, reset command is issued by a reset 
circuit 130 whereby the system 1 10 maintained in a reset state until the temperature of 
the video digitizer returns to within the predetermined temperature range or limit. 

In an embodiment, the FPGA 124 performs a relatively high-rate pixel 
processing in order to unburden the CPU 125 and achieve a desired video processing 
frame rate. This hardware architecture balance reduces overall system cost by removing 
the cost associated with an adequately fast CPU chip. A further frame rate speed up can 
be achieved by using the FPGA and CPU processing simultaneously in parallel. This 
parallel processing is accomplished by FPGA pixel processing the next frame during the 
interval that the CPU is data processing the current frame. Thus, the new FPGA output 
is immediately available to the CPU process when the CPU finishes the current frame 
data processing. This process structure requires the ability to maintain two independent 
sets of data, and is referred to later herein as ping/pong control. 

Turning to FIGURE 16, a top-level simplified block diagram is depicted of 
a processing system in accordance with the present system. The system preferably 
includes an image processing module 21 2 within the FPGA 124ofFIGURE 15,afeature 
processing module 214 executed by the CPU 125 of FIGURE 15, and a detection 
processing module 21 6 also executed by the CPU 1 25 of FIGURE 15. 
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In an embodiment, the image processing module 212 receives stored 
electronic pixel data 218 comprising current image data 220, reference image data 222, 
reference edges data 224, and previous image data 226. Preferably, the current image 
data 220 is the most recently taken image (i.e., taken at t), the previous image data 226 
5 is the next most recently taken image data (i.e., taken at t+1), and the reference image 
data 222 is the oldest of the taken image data (i.e., taken at t+l+x). Moreover, as 
explained in detail further herein, the reference edges data 224 consists of edge data 
extracted from the reference image data 222. 

The image processing module 212 also receives parameter data 228 from the 
10 man-machine interface (i.e., membrane keypad 118 and I/O display board 116 of 
FIGURE 2). As explained in detail further herein, the parameter data 228 includes 
information pertaining what areas (i.e., control zones) that a detected object within the 
field of views (1 12 and 123 of FIGURES 2 and 3) is to result in opening of the door 121 
(FIGURES 2 and 3). 

1 5 As explained in detail further herein, in response to the electronic pixel data 

218 and the parameter data 228, the image processing module 212 produces derived 
image data 230 comprising edge segmentation, motion segmentation, and region 
segmentation. 

The feature processing module 214 receives the derived image data 230 and 
20 the parameter data 228. As explained in detail further herein, the feature processing 
module 214 produces, in response to the image data 230 and parameter data 228, feature 
data 232 comprising edge features, motion features, region features, and frame features. 

The detection processing module 216 receives the feature data 232 and the 
parameter data 228. In response to the data, the detection processing module 216 
25 produces control signals 234 comprising a detection signal for opening and closing the 
door 121 (FIGURES 2 and 3), reference updates, and adaptive thresholds. 

Turning to FIGURES 17 and 1 8, a simplified block diagram is depicted of an 
initialization sequence for the processing system of FIGURE 16. The initialization 
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sequence 234 includes an initialize parameters step 23 6, an initialize threshold tables step 
238, an initialize zones step 240, an initialize FPGA step 242, an initialize video digitizer 
step 244, and an initialize video system step 246. 

The initialize parameters step 236 includes initialization of the man-machine 
5 interface and constant data and derived parameters. During initialization of the man- 

machine interface, user entered data is read and stored into memory. Constant data is 
also loaded into memory along with derived parameters relating to control zones for 
opening and closing the door. 

Thus, as indicated above, upon application of power to the system, the 

10 initialize parameter module 236 initiates the initialization of the man-machine interface 
(i.e., membrane keypad 1 1 8 and I/O display board 1 1 6 of FIGURE 2), constant data, and 
derived parameters. The initialize threshold table module 238 initiates the initialization 
of the area threshold maps from the camera geometry and resolution models. These 
maps are used to determine minimum and maximum pixel characteristics of objects such 

15 as people and vehicles such as, for example, forklifts. 

The initialize zones module 234 initiates the initialization of the control zones 
whereupon data associated with user or predefined safety zones and activation zones is 
complied. The initialize FPGA 242 and the initialize video digitizer 244 initiates the 
initialization of the FPGA 124 (FIGURE 15) and the video digitizer 126 (FIGURE 15), 

20 respectively. In particular, the control status register (CSR) and image buffers pointer 
are initialized during FPGA initialization. Further, the video digitizer is initialized by 
constructing the required instructions and sending them, via the FPGA serial I/O. 

The initialize video system 246 initiates the initialization of the CPU 125 
(FIGURE 15). In particular, the first ping-pong data set is selected. Next, the FPGA is 

25 instructed to capture a video frame. Four images (reference images) are then initialized - 

a grey level reference (R), an edge reference (GER) and an archive edges reference. The 
FPGA is then instructed to process these data. The FPGA outputs are retrieved from the 
FPGA into the Bank 0 database. 
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After the system is initialized, the system operates in a video processing loop 
depicted in the simplified block diagrams of FIGURES 19-22. In an embodiment, the 
video processing loop 250 includes a get next frame step 252, a retrieve FPGA data step 
254, a calculate all features step 214, a calculate detection & control step 258, a save 

5 FPGA data step 260, and a write diagnostics step 262. 

Within the video processing loop 250, the CPU 125 process use the current 
ping/pong buffer to point to, load and unpack that data into a third database - the user 
data - bank U. This data is used in the CPU process later to generate features and 
detection decisions on the current frame. Preferably, at the same time, the CPU process 

10 starts the FPGA capture and process activity on the FPGA 124. While the CPU is 
processing features for the current frame, the FPGA is computing image data for the next 
frame. The detection and control activity sends the safety and activate signals out 
through the FPGA serial I/O interface. The CPU feature and detection processing takes 
longer than the FPGA computations. When the CPU finishes the current frame, the 

15 FPGA data is retrieved to the opposite bank (e.g., Bank 1 if processing Bank 0). 
Diagnostic messages can be output at the end of each frame processing, as well as at any 
point in the video processing. The process then loops to set Bank U to the new current 
bank (Bank 0 or Bank 1), and the FPGA is again initiated. 

Turning to FIGURE 22, a simplified block diagram is provided of an 

20 embodiment of a system resources allocation method, or ping/pong control, in 
accordance with the present invention. As stated previously, the video processing system 
110 includes an FPGA 124 for ipiage processing. This results in the production of 
derived image data230 comprising edge segmentation, motion segmentation, and region 
segmentation. 

25 Once the derived image data 230 is produced, it is preferably stored within 

one of a plurality of memory banks 230a,230b and then provided, via switching, for 
feature processing. Accordingly, the derived image data 230 provided to the feature 
processing module 214 is static. However, the FPGA 124 continuously processes the 
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electronic pixel data 218 and loads the results of the image processing, via switching, 
into the memory bank not currently accessible to the processing module 214. 
Accordingly, the derived image data 230 within the memory banks is accessible to the 
feature processing module via switched between the memory banks 230a,230b on afirst- 
5 in-first-out basis. 

Preferably, two memory banks 230a and 230b are provided. Turning back to 
FIGURE 19, the get next frame step 252 provides for the capture and processing of 
electronic pixel data 21 8 by the image processing module 212 within the FPGA 124. In 
particular, a control and status register (CSR) is used for selection of the memory banks 
10 230a,230b and to set capture & process bit. 

The retrieve FPGA data step 254 provides for obtaining the static data within 
the memory banks for processing of the static data during the calculating all features step 
256. In particular, temporary storage registers and counters are reset, and the static data 
is unpacked to provide the derived image data 230 for processing by the feature 
1 5 processing module 2 1 4 (FIGURE 1 6). 

In an embodiment, and as explained in detail further herein, the feature 
processing module 214 (FIGURE 16) performs the calculate all features step 256, in 
response to the derived image data 230 (FIGURE 1 6) and parameter data 228 (FIGURE 
1 6). In particular, the calculate all features step 256 produces feature data 232 (FIGURE 
20 16) comprising edge or P features, motion features, shadow and lightbeam or region 
features, and frame features. 

Further, the detection processing module 2 1 6 performs the calculate detection 
& control step 258, in response to the feature data 232 (FIGURE 1 6) and parameter data 
228 (FIGURE 1 6). In particular, frame analysis fault flags are evaluated, detection logic 
25 is executed, reference frame update requires are evaluated, automatic adaptive thresholds 
are evaluated, and motion reference data is updated. 

The save FPGA data step 260 occurs once the FPGA 124 (FIGURE 15) 
provides an interrupt to the CPU 125 (FIGURE 15) indicating that the FPGA has 



